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ABSTRACT 


Reynolds  (1976)  described  an  order-analysis  procedure  for 
extracting  un (dimensional  chains  of  items  (or  persons)  from  multi¬ 
dimensional  datasets.  It  is  an  exhaustive  method,  using  one  of 
Cliff's  consistency  indices  to  extract  all  possible  chains  which 
maintain  a  given  level  of  consistency.  A  practical  problem  with  this 
method  is  that  the  number  of  computer  calculations  needed  for  chain 
extraction  can  become  very  high. 

This  paper  outlines  a  procedure  which  uses  graph  theory  to  extract 
nonredundant  chains  efficiently,  rather  than  exhaustively  generating 
all  chains,  as  with  Reynold's  procedure.  It  also  uses  chain  consistency 
as  a  criterion  for  chain  membership.  The  dominance  matrix  can  be 
reconsidered  as  a  labelled  digraph.  From  this  digraph,  all  subgroups 
with  perfect  consistency  are  generated,  and  these  in  turn  are  used  as 
starting  points  in  the  chain  extraction  process.  The  original  dominance 
digraph  is  then  reduced  until  the  chian  is  found  from  each  subgraph. 

This  graph-theoretic  algorithm  may  be  carried  out  using  a  series  of 

J 

matrix  manipulations  performed  on  the  dominance  matrix.  j 
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INTRODUCT I  ON 


Many  psychometricians  have  investigated  the  problem  of  extraction 
of  factors  or  dimensions  from  a  data  matrix.  Ordering  theory  (Airasian 
and  Bart,  1972)  is  one  such  method.  Order  analysis  (Krus,  Bart,  6 
Airasian,  1975)  was  developed  to  examine  the  logic-based  dimensions  in 
binary  data  matrices.  It  makes  use  of  the  dominance  relations  (Coombs, 
196^)  in  data  to  determine  simple  orders. 

Cliff  (1977)  developed  several  order  consistency  indices  from  the 
dominance  relation.  He  shows  his  indices  to  be  comparable  to  classical 
measures  of  internal  consistency  such  as  the  Kuder-Richardson  formula  and 
Loevinger's  index  of  homogeneity. 

Reynolds  (1976)  developed  an  order  analysis  procedure  which  uses 
one  of  Cliff's  consistency  indices  to  recover  all  possible  chains  ( i . e . , 
all  possible  dimensions  that  maintain  a  given  level  of  consistency).  A 
practical  problem  with  this  method  is  that  the  number  of  computer 
calculations  needed  for  chain  extraction  can  easily  become  very  high. 

This  article  Is  concerned  with  presenting  an  efficient  algorithm 
to  extract  all  possible  chains  using  a  genera)  consistency  index.  Our 
extraction  procedure  is  an  interactive  method  using  the  PLATO  computer 
system  at  the  University  of  Illinois.  The  main  algorithm  uses  graph 
theory  to  extract  nonredundant  chains  efficiently,  rather  than 

^The  authors  wish  to  acknowledge  the  contribution  of  Robert  Bail  lie, 
who  wrote  computer  programs  for  the  routines  developed  in  this  paper. 
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exhaustively  generating  ail  chains  as  with  Reynolds'  (1976)  procedure. 

ORDER  ANALYSIS 

The  methods  described  in  this  paper  are  based  on  the  concepts  of 
order  analysis  (Krus,  Bart,  6  Airasian,  1975;  Krus,  1975).  One 
advantage  of  using  order  analysis  is  that  it  is  more  easily  restated 
in  terms  of  graph  theory  concepts  than  are  other  multivariate  procedures 
such  as  factor  analysis  and  cluster  analysis.  In  the  following  section, 
we  will  explain  the  notation  to  be  used  in  describing  the  extraction 
algorithm. 

Simple  Orders 

Mathematically,  simple  orders  are  defined  as  follows.  Let  R 
denote  a  simple  order  relation  between  elements  of  a  set  A.  The 
following  three  properties  hold  for  all  elements  a,  b,  and  c  of  set  A: 

1)  Asymmetry  property 

aRb  implies  bRa,  where  R  means  "not  R". 

2)  Transitivity  property 

aRb  and  bRc  implies  aRc. 

3)  Connectedness  property 

either  aRb  or  bRa. 

An  example  of  simple  order  is  shown  by  the  concept  "less  than".  Let 
A  be  the  set  {I,  2,  3,  4).  Then  the  set  R  of  all  ordered  pairs  of 
elements  from  A  which  maintain  the  relation  "less  than"  is  given  by 
R,  -{(1,2),  (1,3),  (1,4),  (2,3),  (2,4),  (3,4)). 

Directed  Graphs 

Graph  theory  is  a  useful  tool  for  helping  one  to  better 
understand  complicated  sets  of  relationships  (particularly  logical 
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relationships).  We  will  show  that  graphs  can  also  be  useful  in 
developing  algorithms.  We  will  use  directed  graphs  to  extract  elements 
from  sets  which  maintain  the  properties  of  simple  orders. 

Let  us  denote  a  directed  graph  G  by  G  =  (V,E),  where  V  is  the  set 
of  vertices  and  E  the  set  of  edges  (ordered  pairs  of  vertices).  For 
example,  if  we  let  Vj  be  the  set  A  described  earlier, 

G,  =  (VrE,) 

V1  =  A  =  {1,2, 3, 4} 

E]  =  R1  =  {  (1 ,2) ,  (1 ,3) ,  (1 ,4) ,  (2,3) ,  (2,4) ,  (3,4) } 

G^  can  be  represented  graphically,  as  shown  in  Figure  1. 

1 

2 

3 

4 

Figure  1.  Graph  for 

Notice  that  the  three  properties  of  simple  orders  (asymmetry,  transi¬ 
tivity,  and  connectedness)  are  shown  graphically. 

Matrix  Representation  of  Directed  Graphs 

In  order  to  represent  a  directed  graph  G  in  a  computer,  one  commonly 
uses  a  V  *  V  adjacency  matrix  (Deo,  1974)  where  g.^  *  1  or  0  depending  on 
whether  or  not  a  directed  edge  exists  between  vertices  i  and  j  in  V.  The 
matrix  representation  for  G^  is  shown  in  Figure  2. 
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1  2  3  4 

0  111" 
0  0  11 
0  0  0  1 
0  0  0  0 
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Figure  2.  Matrix  Representation  of 


In  testing,  one 
data  matrix.  Figure 


is  typically  dealing  with  a  persons-by- i terns 
3  shows  a  (5x4)  data  matrix,  S. 


1 

2 

Persons  3 

4 

5 


i  terns 

abed 
1111' 
0  111 
0  0  11 
0  0  0  1 
0  0  0  0 


-  S 


Figure  3.  A  Persons  by  Items  Data  Matrix 


The  element  s j j  of  S  is  equal  to  1  if  person  i  gets  item  j  correct, 
and  0  otherwise.  In  order  analysis  terms,  it  is  said  that  when  Sjj  =  1, 
person  I  dominates  item  j  (Krus,  1975).  Conversely,  when  s.j  ■  0,  item 
j  Is  said  to  dominate  person  i.  The  data  matrix  S  is  representative  of 
a  perfect  Guttman  scale.  When  the  items  are  ordered  in  terms  of  difficulty, 
each  person  will  get  each  item  correct  up  to  a  given  difficulty  level, 
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and  no  items  correct  thereafter.  In  actual  testing,  however,  perfect 
Guttman  scales  are  rarely  found. 

Dominance  Matrices 

In  order  analysis,  one  attempts  to  determine  sets  of  items  (or 
equivalently,  persons)  that  exhibit  the  properties  of  simple  orders. 

To  find  these  orders,  one  constructs  a  dominance  matrix  from  the  raw 
data  matrix.  Following  Cliff  (1975)  a  supermatrix  A,  containing  S  and 
S'  (the  transposed  complement  of  S)  as  its  segments,  is  defined.  The 
elements  of  S  and  s'  contain  information  regarding  the  dominances  between 
the  sets  ofpersons  and  items.  The  lack  of  initial  dominance  information 
about  elements  within  each  set  is  denoted  by  zero  matrices  in  the  super¬ 
matrix  A,  which  is  known  as  an  adjacency  matrix. 


A  = 


S' 


0 


However,  we  are  ultimately  concerned  precisely  with  the  dominance 

relations  between  elements  within  sets.  If  we  multiply  A  by  itself,  we 
2 

obtain  the  matrix  A  ,  which  contains  the  within-set  dominance  information 


A 


2 


0 

X 


The  submatrix  N  is  the  item  dominance  matrix.  Element  n^.  of  N  is 

equal  to  the  number  of  persons  for  which  item  i  dominated  item  j  (that  is 

item  i  was  incorrect  and  item  j  was  correct).  Correspondingly,  the 
2 

submatrix  X  of  A  is  the  person  dominance  matrix.  Element  x  equals 
the  number  of  items  for  which  person  s  dominated  person  t  (that  is, 
the  number  of  items  which  person  s  got  correct  and  t  got  incorrect). 
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At  this  point,  the  order  analysis  would  concentrate  on  either  N  or 
X  and  extract  chains  (subsets)  of  items  or  persons  which  exhibit  the 
properties  of  imple  orders  to  a  sufficient  degree  of  approximation. 
The  number  of  chains  needed  to  account  for  all  of  the  elements  in  the 
set  is  interpreted  as  the  dimensionality  of  the  set  (Krus,  1975). 


GRAPHIC  REPRESENTATION  OF  DOMINANCE  INFORMATION 


This  section  outlines  the  graphic  representation  of  several 
order-analysis  concepts  described  in  the  previous  section.  Labelled 
directed  graphs  will  be  used  in  a  later  section  to  develop  an  extraction 
algori thm. 

Item-Person  Dominance  Relations 

Recall  the  binary,  i tems-by-persons  data  matrix  S  from  the  last 
section.  In  Figure  4a  the  arrow  from  i  to  j  denotes  that  person  i 
dominates  item  j  (s.j  =  1).  Figure  4b  shows  that  item  j  dominates 
person  i  (s.j  =  0). 


i  tern  j 


person  i 


item  j 


person  i 


« > 


Figure  4a 

Person  i  Dominates  Item  j 


Figure  4b 

Item  j  Dominates  Person  1 


I  tern- I  tern  Dominance  Relations 

Consider  a  person  X  responding  to  two  items,  a  and  b.  Since 
person  X  can  get  each  item  either  right  or  wrong,  there  are  four  possible 
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response  patterns. 

i  tem 
a_ 

1  1 

1  0 

0  1 

0  0 

Only  the  (1,0)  and  (0,1)  response  patterns  contain  order  information,  as 
shown  by  the  graphs  in  Figure  5.  Paths  of  length  2  are  found  only  for 
these  two  response  patterns.  (  A  path  of  length  2  means  a  succession 
of  two  arrows,  one  starting  at  the  other's  endpoint.) 


0,1) 


a 


b 


0,0) 


a 


b 


(0,1) 


a 


b 


(0,0) 


a 


b 


Figure  5.  Possible  Two  Item  Response 
Patterns  and  their  Graphs 


For  an  entire  persons-by- i terns  data  matrix  S,  the  item-person 
dominances  are  easily  represented  using  a  bipartite  graph.  A  bipartite 
graph  depicts  the  relations  between  the  elements  of  one  set  and  the 


elements  of  another  set.  Figure  6  shows  a  data  matrix,  Its  adjacency 
matrix,  and  its  corresponding  bipartite  graph. 


A  6  C  1  2  3  4 


0 

0 
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,  1 

0 

1 

0 

0 

0 

1 

,  o 

1 

0 

0 

0 

0 

1 

,  0 

0 

I 

0 

1 

1 

f 

,  0 

0 

0 

1 

0 

1 

1 

,  0 

0 

0 

0 

1 

0 

1 

,  0 

0 

0 

0 

0 

1 

1 

1  0 

0 

0 

1 

1 

0 

0 

0 

0 

0 


Figure  6.  A  Data  Matrix  with  its  Adjacency 
Matrix  and  Bipartite  Graph 
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The  Item  dominance  matrix  for  the  data  matrix  In  Figure  6  can  also 
be  shown  by  use  of  a  labelled  graph.  Figures  7a  and  7b  show  this 
dominance  matrix  and  Its  graph.  An  element  n^.  represents  the  number 
of  ij  dominances.  An  element  n^.  j  represents  the  number  of  jl  dominances, 
and  can  also  be  thought  of  as  the  number  of  ij  counterdom I nances.  In 
the  graph,  the  numerals  in  parentheses  are  related  to  these  dominances 
and  counterdominances  as  follows:  The  second  number  is  the  number  of 
dominances  in  the  direction  of  the  referenced  arrow,  and  the  first 
number  is  the  sum  of  the  dominances  and  counterdominances. 


A 

B 

C 


ABC 
0  2  2 
1  0  2 
0  1  0 


Figure  7a.  Item  Dominance 
Matrix  for  Data  Matrix  In 
Figure  6. 


Figure  7b.  Labelled  Graph 
for  Item  Dominance  Matrix 
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CONSISTENCY  INDEX 


We  define  a  general  consistency  index,  C,  as 


C  =  a(U  /U)  -  e 

a 

where  U  ■  E  E  n..  (the  total  number  of  item  dominances) 

j  k  Jk 

U  *  E  E  n..  (the  number  of  dominances  above  the  main  diagonal) 
3  j  k>j  ->k 

o  and  0  are  parameters  which  determine  the  scale  and  origin, 
respectively,  of  C, 


For  a  given  data  matrix,  the  total  number  of  dominances  (U)  is 
fixed,  while  U  Is  dependent  on  the  order  of  the  items.  When  the  items 

3 

are  ordered  in  terms  of  difficulty,  U  is  maximized. 

When  a  «*  2  and  0  *  1,  C  is  equivalent  to  the  index  used  by 

Reynolds  in  his  exhaustive  method.  This  is  also  equivalent  to  CfJ , 

one  of  Cliff's  consistency  indices  (Cliff,  1977),  since 

2U  2U-U  U  -  (U-U)  U  -U.  v 

L  “  U  "  ‘  51  U  “  U  -  — g  =  v  -  ttl 


where  U. 


E  E  n 
j  k>j 


jk 


(the  number  of  dominances  below  the  main  diagonal) 


m 


E  E  (n..  -  n.  .) 


j  k>j 

E  E  n.. 
j  k  Jk 


Jk  kj 


It  is  also  easy  to  show  that  with  a  =  4  and  0  ■  3»  C  becomes 
Ct2»  another  index  proposed  by  Cliff. 
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A1 thought  the  above  equation  for  C  Involves  two  parameters, 
a  and  0  ,  It  Is  seen  that  0  Is  a  function  of  a  by  considering  a  dataset 
which  forms  a  Guttman  simplex  so  that  U  =  U  and  C  =  1 .  Thus, 

d 

1  =  a(l)  -  0,  or  0  **  a  -  1.  Hence 
C  =  a (U  /U )  -  (a  -  1) 

a 

Therefore,  we  have  only  one  parameter  a,  and  its  value  is  determined 
so  that  C  wi 1 1  be  approximately  equal  to  zero  for  a  dominance  matrix 
based  on  a  random  set  of  data.  With  appropriate  choices  of  a,  C  can 
approximate  Cliff's  Ctj,  C^,  or  C^. 

Ua  C  +  8 

Let  C'  =  jp  =  .  Then  : 


U 


C'  = 


a _ 

U  U 


u 


a  +  Ub  "  2  E 


*  2  nik 

l±>i 

—  -L-  V 


2  2  d.. 

J  k>j  Jk 

’  (c„  T 


2  2  n.,+2  2  n . ,  2  2 

j  k>j  JK  j  k<j  lk  j  k<j 


*  E  njk 

_ l'k>j  Jk 

2  2  (r ~ 


%  +  "jk* 


2  2  ( C  .  +  d  ) 

j  k<j  Jk  Jk 


where  d^k  Is  the  number  of  dominances  from  j  to  k. 

Cj|^  is  the  number  of  counterdominances  from  j  to  k  (=n^j ) 
Or  we  can  say 


2  2  d., 

J  k>j  -«k 


2  2  d 

j  k<j 


Jk 


2  2  d.. 

i-K>J  Jk 

2  2a., 

j  k>j  kJ 


where  ajk  -  cjk  +  djk< 
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Krus  (1975)  showed  that  dominance  matrices  can  be  split  Into 
additive  matrices  according  to  Item  or  person  subsets.  From  this, 
letting  each  person  be  treated  as  a  subset,  we  have 


C' 


n  n  N 

EE  £ 


J  k>j 
n  n  N 
£  £  E 

j  k<j  1-1 

N  n  n 

E  E  E 

hli  A>J 

N  n  n 

E  E  E 
1-1  J  k<j 


1  =  1 


*{'> 

jk 


Jk 


s 


Ik 


’ij 


+  s' 


J  i 


sik> 


N 

E 

1-1 

N 

E 

1  =  1 


n  n 

E  E 

iJSLl 

n  n 
E  E 
j  k<j 


N 

E 

1  =  1 
N 

E 

1=1 


d 


a 


(1) 

jk 


0) 

jk 


This  general  consistency  Index,  C,  has  the  advantage  of 
additivity  over  the  above-mentioned  indices  C^,  C^,  and  C^,..  Each 
item  or  person  can  be  thought  of  as  having  an  Individual  C  value.  When 
groups  of  items  or  persons  are  combined,  both  the  numerator  and 
denominator  of  the  U  /U  term  of  C  change  by  an  additive  amount.  This  is 

d 

not  the  case  for  Ctj,  C^,  or  which  are  further  influenced  by 

"chance"  consistency  in  a  non-additive  fashion. 

In  the  next  section  we  utilize  this  additive  property  to  extract 


all  possible  uni  dimensional  chains  In  a  more  efficient  fashion. 
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EXTRACTION  PROCEDURE 

Our  extraction  procedure  is  an  interactive  method  using  the  PLATO 
computer  system  at  the  University  of  Illinois.  The  main  algorithm  uses 
graph  theory  to  extract  nonredundant  chains  efficiently,  rather  than 
exhaustively  generating  all  chains,  as  with  Reynolds'  (1976)  procedure. 

It  uses  the  consistency  index  C  as  a  criterion  for  chain  membership, 
taking  into  account  the  fact  that  the  numerator  and  denominator  both 
maintain  additivity  under  addition  and  deletion  of  elements  in  a  chain. 

The  dominance  matrix  can  be  interpreted  as  a  labelled  digraph. 

From  this  graph,  all  subgroups  with  perfect  consistency  are  generated, 
and  these  in  turn  are  used  as  starting  points  in  chain  extraction.  Given 
a  criterion  consistency  value  Cq,  the  original  dominance  digraph  is  then 
reduced  until  a  chain  is  found  from  each  subgraph.  After  extracting  all 
possible  chains,  redundant  chains  can  be  eliminated  and  Cq  can  be  changed 
to  select  more  or  fewer  chains,  as  desired. 

As  a  backdrop,  we  first  briefly  review  Reynolds'  (1976)  method, 
which  extracts  all  possible  chains  which  satisfy  the  condition  that 
criterion  C  (which  is  equivalent  to  Cliff's  consistency  index  C^) 
exceed  some  cutoff  value.  His  approach  is  an  iterative  method  generating 
all  chains  starting  at  each  item. 

For  each  item,  k,  a  consistency  value  is  calculated  for 

combining  the  new  item  with  items  already  in  the  chain.  The  item 
which  produces  the  smallest  decrement  in  is  added  to  the  chain, 

producing  Cp  : 

C'  - 
P 


max  C  ..  , 

k  p+k 


Thfs  pro-ess  of  adding  the  "most  consistent"  Item  Is  repeated  using 
the  remaining  I  terns  until  the  overall  chain  consistency  drops  below 
the  criterion  consistency  value.  At  this  point  the  procedure  stops 
and  the  chain  is  complete.  This  method  is  reapplied  using  each  item 
as  a  starting  point  so  that,  for  m  Items,  m  chains  are  extracted. 
Duplicate  chains  are  deleted,  leaving  a  unique  set  of  chains  that  are 
Interpreted  as  the  dimensions  of  the  data  matrix  (Reynolds,  1976). 

Reynolds'  procedure  tolerates  a  lot  of  redundancy  in  order  to 
ensure  that  all  chains  are  extracted.  That  is,  for  each  item  in  a 
given  chain,  using  it  as  a  starting  point  will  frequently  result  in 
the  re-production  of  an  already  existing  chain. 

The  total  number  of  possible  consistency  calculations  for  a 
k-item  data  matrix  can  be  computed.  Starting  at  each  item  there  are 

(k-1)  +  (k-2)  +  ...  +  (k-k-1)  -  k^~H 

calculat ions.  Summed  across  starting  points  (items),  we  have  a  total  of 

k  x  k(k-t>  .=  k2(k-l) 

K  2  2 

calculations,  which  is  of  the  order  of  k^.  As  k  increases  the  number 
of  calculations  needed  goes  up  rapidly.  For  instance,  for  a  20-item 
data  matrix,  as  many  as  3800  calculations  may  be  needed.  Many  of 
these  calculations  are  unnecessary. 

fc  *  * 

Our  algorithm,  on  the  other  hand,  begins  by  determining  all 
subsets  of  items  with  perfect  consistency  (01.0);  that  is,  subsets 
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which  satisfy 


C 

P 


Z 

Pel 

Z 

pel 


d. 

t 


a . 


1.0 


where  p  denotes  any  chain  of  items  with  C  =  1.  (A  chain  consisting 
of  only  one  item  is  permissible.)  Note  that  there  will  likely  be 
more  than  one  starting  item  chain.  At  this  point,  items  are  added 
one  by  one  to  a  given  chain  in  such  a  way  that  ■ ts  cons i stency  stays 
as  high  as  possible.  When  its  value  drops  to  some  predetermined  Cq, 
we  stop  adding  further  items  to  this  chain,  but  select  another  chain 
and  repeat  the  procedure. 

We  have  considered  three  different  strategies  for  deciding  on 
the  successive  items  to  add  to  a  chain: 

(1)  Take  each  one  of  the  remaining  items  in  turn,  calculate  the 
new  consistency  index  when  each  is  added  to  the  chain,  and  select  that 
item  which  yields  the  largest  C  value.  This  is  essentially  the  same 
as  Reynolds'  (1976)  procedure,  and  it  is  time  consuming  because 
repetitious  calculations  have  to  be  done  for  each  item  in  turn. 

(2)  Without  taking  the  chain  as  a  whole  into  consideration, 

look  at  the  "individual"  consistency  index 
d. 


C. 


ind 


ai 


(i  t  I) 


of  each  remaining  item,  and  choose  the  item  with  the  largest  C 


1  nd 


to  add  to  the  chain.  (Note,  however,  that  each  time  an  item  is  added 
to  a  chain,  the  individual  consistency  index  for  each  of  the  remaining 
items  has  to  be  calculated  anew.) 
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(3)  Out  of  the  remaining  items,  take  the  three  with  the  highest 
individual  consistency  values,  and  add  each  one  of  these,  in  turn,  to 
the  chain.  Choose  the  one  that  produces  the  smallest  decrease  in  the 
overall  chain  consistency  to  actually  add  to  the  chain.  (This 
method  is  to  allow  for  the  possibility  --  which  we  have  not  experienced 
so  far  —  that  the  item  with  the  highest  individual  consistency, 
selected  by  method  (2),  may  not  necessarily  produce  the  smallest 
decrease  in  the  chain  consistency.) 

Our  computer  algorithm  can  use  any  of  the  three  strategies  just 
described;  moreover,  even  when  method  (1)  is  used,  we  have  made 
arrangements  to  keep  the  repetitious  calculations  at  a  minimum.  We 
shall  now  describe  our  algorithm  in  detail,  giving  an  example  using 
the  second  strategy. 

Chain  Extraction  Algorithm 

1.  Start  with  a  persons  x  items  data  matrix. 

1.1  Compute  the  dominance  matrix. 

N  *  [  nij  J  k,j  -  1,2 . m 

Note  that  this  matrix  is  not  a  binary  matrix  and  that  n ^  j  =0. 

Reorder  the  dominance  matrix  in  terms  of  item  difficulties. 

1.2  Construct  a  matrix  0  such  that  each  element  In  Its  lower 
triangle  is  equal  to  the  sum  of  the  corresponding  element  of  N 
plus  the  symmetric  element  of  N.  The  upper  triangle  elements  are 
equal  to  the  corresponding  element  of  N  and  the  main  diagonal 
elements  are  al 1  zero.  That  Is, 
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0 


where  o. . 

U 


i,j  =  1 ,2,.. . ,m 


i  <  j 
i  =  j 
i  >  j 


2.  Graph  Initialization 

2.1  Construct  a  labelled  graph  from  matrix  0. 

G  =  (V. E) 

V  =  {l,2,„..,m  }  (the  total  set  of  items) 

E  =  {  ( i , j )  l  =  l,2,...,m;  j  =  i  +  l,!+2 . m  } 

2.2  Initialize  labels  for  vertices  and  edges: 

e_  =  (a^.j,  d_)  =  (Oj.,0.  J  for  i=l,2,...,m;  j«i+l ,1+2,. .. ,m 

v  j  =  [p j  »q  j]  -  [0,0]  for  i-1,2 . m 

The  graph  labels  refer  to  the  consistencies  both  between  vertices 
and  "within"  vertices.  As  the  algorithm  proceeds,  some  of  the 
vertices  will  be  merged  to  form  new  vertices.  Each  v.  will  refer  to 
the  consistency  of  the  set  of  Items  within  vertex  i.  Hence,  each 
p.  is  equal  to  the  sum  of  the  elements  in  the  lower  triangle  of  0 
that  correspond  to  the  dominances  between  elements  contained  in 
vertex  1.  Likewise,  q.  is  equal  to  the  sum  of  the  corresponding 
elements  of  the  upper  triangle  of  0.  Thus,  each  v^  is  initialized 
at  [0,0]  . 

2.3  Initialize  a  chain  matrix  (CH) 

This  will  start  out  as  an  m  x  m  binary  matrix  that  will  contain 
the  item  chains.  All  elements  of  CH  will  be  initialized  at  zero 
except  for  the  main  diagonal  elements,  which  are  equal  to  one.  When 


I 
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the  algorithm  is  finished,  ch.j  will  equal  one  if  item  j  is  a 
member  of  chain  I;  otherwise,  it  will  beequal  to  zero.  Also, 
redundant  chains  will  be  eliminated  during  the  extraction  process 
so  that  the  final  number  of  chains  (rows)  in  CH  will  be  less  than  m. 

3.  Extraction  of  maximal  chain  with  a  consistency  greater  than  a 

criterion,  C'  . 

o 

3. 1  For  a  given  chain  L,  merge  the  vertices  and  edges  included  in 
the  chain.  For  vertices  i  and  j  (i<j)  where  ch^.^l  and  ch^=0, 
merge  vertices  1  and  j  to  form  vertex  i • j  with  edges 


(k, i * j )  k  <  i 
( i  *  j  ,  k)  i  <  k 


for  all  k  e  V-{ i , j } . 


The  new  labels  in  the  graph  are: 

v,  •  -  Tp.  +  p.  +  a..,  q.  +  q.  +  d,  .1 
i  »j  L  ?  j  j  i  ’  N  u  i  jJ 


k,  i  •  j 


( aik +  >>  dki  *  dkj  1 


k  <  i 


ei.j,k  =  (  aki  +  ajk’  dik  +  dkj  >  '  <  k  "  jf  for  a11  kE  V"{i’j}< 


i*  j  ,k 


{  aki  +  akj*  d,k  +  djk}  k  >  J 


3.2  Look  for  the  candidate  vertex  (item)  which  should  be  merged  next. 
Find  the  item  with  the  largest  "individual"  consistency  (strategy  2) 


^®k,  i  *j^2 


Lek, i  -  j3 1 

tei » i  .k-^2 

Ce  i  •  j  ,  kl  1 


k  <  i 


k  >  i 


where  [efl  ^  and  [eg  are  the  first  and  second  components  of 

e  . ,  and  i»j  represents  items  already  in  the  chain. 

3  i  D 


i 
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3.3  Compute  the  new  consistency  (C1)  with  the  best  candidate  item, 
k,  added  to  the  chain. 


q ,  .  +  d ,  .  . 
i  =  HJ*1  t-lik 

k’J*j  pi • j  +  ak, i • j 


or 


C'  .  . 


q.  .  +  d.  ,  . 

i‘J  k,i»j 


?  •  j ,  k  p.  .  +  a .  .  . 

J*  Ki*J  i  *  J  ,  k 


k  <  i 


k  >  i 


3.4  If  C'  >  C1  then  add  vertex  k  to  the  chain  and  merge  the  graph. 
-  o 

Return  to  step  3-2  . 

If  C1  <  C*  ,  the  algorithm  is  finished. 
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AN  ILLUSTRATIVE  EXAMPLE 


We  will  now  illustrate  the  use  of  our  algorithm  using  the  example 
given  in  Reynolds  (1976,  p.  24) .  The  consistency  index  he  used  is 
equivalent  to  our  general  consistency  index  with  parameters  a  =  2  and 


6*1. 

1 .  Data  Matrix 

1.1  The  ordered  item  dominance  matrix  for  the  six  items  is: 


N  =  S*  x  S 


F 

D 

E 

C 

A 

B 


D 

3 

0 

2 

0 

2 

0 


E 

2 

2 

0 

2 

1 

1 


CAB 
4  4  5 
1  4  3 
3  3  4 
0  4  2 
3  0  3 
0  2  0 


1.2  The  corresponding  0  matrix  is: 


0  = 


F 

D 

E 

C 

A 

B 


D 

3 
0 

4 
1 
6 
3 


E  C  A  B 

2  4  4  5' 
2  14  3 
0  3  3  4 

5  0  4  2 

4  7  0  3 

5  2  5  0 


2.  Graph  Initialization 

2,1,  2.2  The  labelled  graph  corresponding  to  matrix  0  is  shown  below 


in  Figure  8 
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Figure  8.  Labelled  graph  of  matrix  0. 
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2.3 


Initialize  the  chain  matrix 


CH  = 


(CH). 
D  E 
0  0 
1  0 
0  1 
0  0 
0  0 
0  0 


CAB 
0  0  0 
0  0  0 
0  0  0 
1  0  0 
0  1  0 
0  0  1 


3.  Extraction  of  Perfect  Item  Chains  (C  =  1.0)  . 

3.1,  3.2  Select  vertices  with  perfect  consistency  (C  =  1.0)  for 
each  chain,  and  delete  redundant  chains. 


CH  = 


D 

0 

1 

0 

1 

0 

0 


E 

1 

0 

1 

0 

0 

0 


C  A 
0  0 
1  0 
0  0 
1  0 
0  1 
0  0 


B 

0 

1 

0 

1 

0 

1 


reduce 


CH  = 


D 

0 

1 

0 

0 


E  C  A  B 
10  0  0 
0  10  1 
0  0  10 
0  0  0  1 


Repeat  step  3.  for  a  lower  criterion  consistency. 

4.  Extraction  of  Maximal  Chain  greater  than  a  Criterion  Consistency  (Co=.8). 
4.1  Merge  vertices  and  edges  included  in  the  chain. 

We  get  Figure  9.  Note  that  chain  1  is  being  processed. 
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Figure  9.  Merging  of  Vertices  for  Chain  1 


4.2  Determine  the  next  candidate  item. 

Of  these  5/8,  7/10,  7/8,  and  9/10,  the  value  9/10  is  the  largest. 
Therefore  the  vertex  B  Is  the  candidate. 

4.3  Compute  C  (a  ■  2,  B  ■  1)  for  the  chain  with  the  candidate  added. 
C  -  2(2+9)/ (2+10)-l  -  .8333 

4.4  If  C  >  CQ  then  add  candidate  item  to  chain,  merge  the  graph  and 

return  to  step  3.2.  If  C  <  C  ,  do  not  add  candidate  item  and  chain 

o 

is  complete.  C  (*.8333)  >  CQ  (“.8)  so  we  add  the  candidate,  merge 
the  graph  and  get  CH  matrix  and  Figure  10. 
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F  D  E  C  A  B 

1  p  0  1  0  0  1 

CH  -  2  0  10  10  1 

5  10  0  0  1  0 

6  10  0  0  0  1 


Figure  10„  Merging  of  Vertex  F,  E  and  B 

We  repeat  the  4.2  through  4.4.  This  time  vertex  A  becomes  the 

candidate  because  10/13  Is  the  largest  value,  but  the  new 

C  -  2(1 1+10)/(12+13)-1  =  .68  <  C(=.8),  so  the  algorithm  then  stops. 

Repeating  the  process  for  the  other  chains,  we  get  the  final  chain 

matrix  (C  ■  .8). 
o 

F  0  E  C  A  B  .5161 

1  p  0  1  0  0  1  1  .8333 

CH  -  2  110101  .8000 

5  101010  .8000 
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The  value  In  the  upper  right-hand  corner  (.5161)  Is  the  C  value  for 
a  chain  containing  all  of  the  items.  Note  that  each  Individual 
chain  represents  an  improvement  over  .5161. 

The  interactive  procedure  to  extract  chains  is  summarized  in 
Figure  11.  Each  block  consists  of  one  or  more  routines  which  are 
implemented  on  the  PLATO  system  at  the  University  of  Illinois. 
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Figure  11.  Extracting  procedure  on  PLATO  system 
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DISCUSSION 

We  Introduced  a  general  consistency  Index  In  this  paper  such  that, 
with  parameters  values  a  ■  2  and  g  ■  1,  It  Is  equivalent  to  the  index 
which  Reynolds  (1976)  used  In  his  article,  and  also  to  one  of  Cliff's 
indices,  C^  (1977).  With  a  ■  4  and  g  =  3  It  becomes  Ct2»  another  index 
by  Cliff.  With  appropriate  choice  of  a  (and  g  •«  a-1)  we  can 
approximate  others  of  Cliff's  Indices,  C^,  C^,  and  C^,.. 

One  important  feature  of  the  general  consistency  index  is  that 
additivity  holds  in  both  numerator  and  denominator.  It  enables  us 
to  use  a  graphical  merging  technique.  As  a  result  we  can  give  an 
efficient  algorithm  to  get  all  possible  chains,  which  were  obtained 
by  Reynolds  by  his  exhaustive  method. 

In  the  current  paper  we  have  picked  up  the  I  terns  in  order  to  get 
item  chains.  But  we  can  easily  extend  the  technique  described  here  to 
a  method  that  both  picks  up  items  and  eliminates  persons  In  order  to 
get  highly  consistent  item  chains,  because  additivity  in  numerator  and 
denominator  holds  for  persons  as  well  as  items.  In  some  cases  a 
few  persons  "contaminate"  our  data,  therefore  elimination  of  these 
persons  is  a  good  way  to  get  good  Item  chains.  Our  efficient  algorithm 
ensures  the  extracting  of  chains  for  a  large  number  of  items  and 
persons.  We  can  also  apply  the  same  technique  to  get  consistent 
person  chains,  by  picking  up  or  eliminating  items  or  persons. 
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1  Ted  M.  I.  Yellen 

Technical  Information  Office,  Code  201 
NAVY  PERSONNEL  RAD  CENTER 
SAN  DIEGO.  CA  92152 

t  Library ,  Code  PZOIL 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

5  Technical  Director 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Director,  Navy  Personnel  RAD  Center 
Washington  Llason  Office 
Building  200,  2N 
Washington  Navy  Yard ,  DC  20374 

6  Commanding  Officer 
Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

1  Psychologist 

0NR  Branch  Office 
Bldg  114,  Section  D 
666  Summer  Street 
Boston,  MA  02210 

1  Psychologist 

0NR  Branch  Office 
536  S.  Clark  Street 
Chicago,  IL  60605 

1  Office  of  Naval  Research 
Code  437 

800  N.  Quincy  SStreet 
Arlington,  VA  22217 

S  Personnel  A  Training  Research  Programs 
(Code  456) 

Office  of  Naval  Research 
Arlington,  VA  22217 


1  Psychologist 

ONR  Branch  Office 
1030  East  Green  Street 
Pasadena,  CA  9110) 

1  Scientific  Director 

Office  of  Naval  Research 
Scientific  Liaison  Croup/Tokyo 
American  Embassy 
APO  San  Francisco,  CA  96503 

1  Special  Asst,  for  Education  and 
Draining  (OP-01E) 

Ms.  2705  Arlington  Annex 
Washington.  DC  20370 

1  Office  of  the  Oiler  of  Naval  Operations 
Research,  Development,  and  Studies  Branc 
(OP-102) 

Washington,  DC  20350 

1  Long-Range  Manpower,  Personnel,  and 

Training  Planning  Branch  (0P-110) 
Room  G828 
Arlington  Annex 
Washington,  DC  20350 

1  Head,  Manpower  Training  and  Reserves 
Section  (Op-964D) 

Room  4A478,  The  Pentagon 
Washington,  DC  20350 

1  Captain  Donald  F.  Parker,  USH 
Commanding  Officer 
Navy  Personnel  RAD  Center 
San  Diego.  CA  92152 

1  LT  Frank  C.  Petho,  MSC,  USN  (Ph.D) 

Code  L51 

Naval  Aerospace  Medical  Research  Laborat 
Pensacola,  FL  32508 

1  Dr.  Gary  Poock 

Operations  Research  Department 
Code  55PK 

Naval  Postgraduate  School 
Monterey,  CA  93940 

1  The  Principal  Deputy  Assistant 

Secretary  of  the  Navy  (MRAAL) 

4E780,  The  Pentagon 
Washington,  DC  20350 

1  Director,  Research  A  Analysis  Division 
Plans  and  Halley  Department 
Navy  Recruiting  Command 
4015  Wilson  Boulevard 
Arlington,  VA  22203 

1  Dr.  Worth  Scanland 

Chief  of  Naval  Education  and  Trafning 
Code  N-5 

NAS,  Pensacola,  FL  32508 

1  Dr.  Robert  C.  Smith 

Office  of  Chief  of  Naval  Operations 
OP-987 H 

Washington,  DC  20350 

1  Or.  Alfred  F.  Mi  ode 
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(TAEG) 

Dept,  of  the  Navy 
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12  Defense  Oocunentatlon  Center 
Cameron  Station,  Bldg.  5 
Alexandria.  VA  2231“ 

Attn:  TC 

1  Dr.  Dexter  Fletcher 

ADVANCED  RESEARCH  PROJECTS  AGENCY 
1400  WILSON  BLVD. 

ARLINGTON,  VA  22209 
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Testing  Directorate 
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Washington.  DC  20301 
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OfTice  of  the  Under  Secretary  of  Defense 
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Washington.  DC  20301 
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of  Defense  (MRA41) 

3B930  The  Pentagon 
Washington,  DC  20301 
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Washington,  DC  20208 

1  Dr.  Lorraine  D.  Eyde 
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Office  of  Personnel  Management  of  USA 
1900  EStreet  NW 
Washington,  D.C.  20415 

1  Jerry  Lehnus 

REGIONAL  PSYCHOLOGIST 

U.S.  Office  of  Personnel  Management 

230  S.  DEARBORN  STREET 

CHICAGO,  IL  60604 

1  Dr.  Joseph  I.  Ltpson 
SC  DR  W-638 

National  Science  Fouidatlon 
Washington,  DC  20550 

1  William  J.  Me  La  ur  In 
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1  Dr.  Andrew  R.  Molnsr 
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Washington,  0C  20550 
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Office  of  Personnel  Manegment 
1900  E  Street  NW 
Washington.  DC  20415 


1  Dr.  H.  Wallace  Slnaiko 
Program  Director 

Manpower  Research  and  Advisory  Services 
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1  Dr.  Vern  W.  Urry 

Personnel  R4D  Center 
Office  of  Personnel  Management 
1900  E  Street  NW 
Washington,  DC  20415 

1  Dr.  Frank  Withrow 

U.  S.  Office  of  Education 
400  Maryland  Ave.  SW 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
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National  Science  Foundation 
Washington,  DC  20550 
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1  Dr.  Erllng  B.  Andersen 
Department  of  Statistics 
Studlestraede  6 
1455  Copenhagen 
DENMARK 

1  Dr.  John  R.  Anderson 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Anderson,  Thomas  H.,  Ph.D. 

Center  for  the  Study  of  Reading 
174  Children's  Research  Center 
51  Gerty  ft-lve 
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1  Dr.  John  Annett 

Department  of  Psychology 
University  of  Warwick 
Coventry  CV4  7AL 
ENGLAND 

1  DR.  MICHAEL  ATWOOD 
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40  DENVER  TECH.  CENTER  WEST 
7935  E.  PRENTICE  AVENUE 
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1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
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1  Dr.  Alan  Baddeley 

Medical  Research  Cornell 

Applied  Psychology  Unit 
15  Chaucer  Road 
Cambridge  CB2  2EF 
ENCLAND 

1  Dr.  Patricia  Baggett 
Department  of  Psychology 
University  of  Denver 
University  Park 
Denver,  CO  80208 
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Educational  Testing  Service 
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1  DezWps  1m  Streitkraefteamt 
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WEST  GERMANY 

1  Dr.  Nicholas  A.  Bond 
Dept,  of  Psychology 
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600  Jay  Street 
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1  Dr.  Lyle  Bourne 

Department  of  Psychology 
University  of  Colorado 
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1  Dr.  Robert  Brennan 

American  College  Testing  Programs 

P.  0.  Box  168 
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UNIVERSITY  PLAZA.  SUITE  10 
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Department  of  Administrative  Sciences 
U.  S.  Naval  Postgraduate  School 
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1  DR.  MARTIN  F.  WISKOFF 
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SAN  DIEGO.  CA  92152 
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U.  S.  Army  Research  Institute  for  the 
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5001  Elsenhower  Avenue 
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U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
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U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
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Army  Research  Institute 
5001  Eisenhower  Blvd. 

Alexandria,  VA  22333 
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5001  EISENHOWER  AVENUE 
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U.S.  Army  Research  Institute 
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j _ 
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Attn:  PERI-OK 
Army  Research  Institute 
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1  LTC  Michael  Plunmer 
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U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
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Alexandria,  VA  22333 
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U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
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US  Army  Institute  of  Administration 
Attn:  Dr.  Sherrill 
FT  Benjamin  Harrison,  IN  *6256 

1  Dr.  Frederick  Stelnheiaer 

U.  S.  Army  Reserch  Institute 
5001  Elsenhower  Avenue 
Alexandria.  VA  22333 

J  Dr.  Joseph  Ward 

U.S.  Army  Research  Institute 
5001  Elsenhower  Avenue 
Alexandria,  VA  22333 
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1  Air  Force  Human  Resources  Lab 
AFHRL/MPD 

Brooks  AFB,  TX  78235 

1  U.S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NL 
Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Dr.  Earl  A.  Allulsl 
HO.  AFHRL  (AFSC) 

Brooks  AFB.  TX  78235 

1  Dr  .  Phil  lp  De  Leo 
AFHRL/TT 

Lowry  AFB.  CO  80230 

1  Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
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Bolling  AFB,  DC  20332 

1  Research  and  Measurment  Dlvis*^ 
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Randolph  AFB,  TX  781 08 

1  Dr  .  Malcolm  Ree 
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Brooks  AFB.  TX  78235 


1  Dr.  Marty  Rockway  (AFHRL/TT) 
Lowry  AFB 
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1  Dr.  Frank  Schufletowski 
U.S.  Air  Force 
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Marines 


1  H.  William  Greenup 

Education  Advisor  ( EO 3 1  > 

Education  Center,  MCDEC 
Ouantico.  VA  2213* 

1  Director,  Office  of  Manpower  Utilization 
HQ,  Marine  Corps  (MPU) 

BCB,  Bldg.  2009 
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Code  MPI-20 
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1  Special  Assistant  for  Marine 
Corps  Matters 
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Office  of  Naval  Research 
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Arlington.  VA  22217 
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Washington.  DC  20380 
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HO,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20380 
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Oklahoma  City,  OK  73169 
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I  Or.  William  E.  Corrmtn 

Director.  Iowa  Testing  Programs 
33*  Lindquist  Center 
University  of  Iowa 
Iowa  City.  IA  522*2 

1  Dr.  Allan  M.  Collins 

Bolt  Beranek  A  Newnan.  Inc. 

50  Moulton  Street 
Cambridge.  Ma  02138 

1  Dr.  Meredith  P.  Crawford 

American  Psychological  Association 
1200  17th  Street.  N.W. 

Washington,  DC  20036 

1  Dr  .  Hans  Cronbag 

Education  Research  Center 
University  of  Leyden 
Boerhaavelaan  2 
233*  EH  Leyden 
The  NETHERLANDS 

1  ERIC  Facll ity-Acqulsltlons 
*833  Rugby  Avenue 
Be the sd a.  M0  2001* 

1  Dr.  Leonard  Feldt 

Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City.  IA  522R2 

I  Dr.  Ricfiar^  L.  Ferguson 

The  American  College  Testing  frogram 

P.0.  Box  166 

Iowa  City.  IA  522*0 

1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Unlv.  Prof.  Dr.  Gerhard  Fischer 
Lleblggasae  5/3 
A  1010  Vienna 
AUSTRIA 

1  Professor  Donald  Fitzgerald 
University  of  New  England 
Armidalc.  New  South  Wales  2351 
AUSTRALIA 

1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  frgan. 
Suite  900 

*330  East  West  Highway 
Washington,  DC  2001* 

1  Dr,  John  R.  Frederiksen 
Bolt  Beranek  A  Hetman 
50  Moulton  Street 
Cambridge,  HA  02138 

1  ^g^lMERf  GLASER 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  1521 3 

1  Dr .  Ross  Green 
CTB/HcGraw  Hill 
Del  Monte  Research  Park 
Monterey.  CA  939*0 


1  DR.  JAMES  G.  GREEN0 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH.  PA  15213 

1  Dr  .  Ron  Hambleton 
School  of  Education 
University  of  Massechusetts 
Amherst,  MA  01002 

1  Dr.  Chester  Harris 
School  of  Education 
University  of  California 
Santa  Barbara,  CA  93106 

1  Dr.  Lloyd  Humphreys 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Library 

Hum RR0/ We stern  Division 
27857  Berwick  Drive 
Carmel.  CA  93921 

1  Dr.  Steven  Hunka 

Department  of  Education 
University  of  Alberta 
Edmonton,  Alberta 

CANADA 

1  Dr.  Earl  Hunt 

Dept .  of  Psychology 
University  of  Washington 
Seattle,  WA  93105 

1  Dr  .  Huynh  Huynh 

College  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  Dr.  Douglas  H.  Jones 
ft*  T-255 

Educational  Testing  Service 
Prinopton,  N J  08*50 

3  Journal  Supplement  Abstract  Service 
American  Psychological  Association 
1200  17th  Street  N.W. 

Washington,  DC  20036 

1  Dr.  Arnold  F.  Kanarlck 
Honeywell,  Inc. 

Honeywell  Plaza  MN12-3166 
Minneapolis,  MN  55*08 

1  Professor  John  A.  Keats 
University  of  Newcastle 
AUSTRALIA  2}Q0 

1  Mr.  Marlin  Kroger 
1117  Via  Go  let a 

Palos  Verdes  Estates,  CA  9027* 

1  Dr.  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Alan  lesgold 
Learning  R&D  Center 
University  of  Pittsburgh 
Pittsburgh,  PA  15260 


1  Dr.  Michael  Levine 

210  Education  Building 
University  of  Illinois 
Champaign.  IL  61820 

1  Dr.  Charles  Lewis 

Facultelt  Sociale  Wetenschappen 
Ri jksuniversitelt  Groningen 
Oude  Boter lngeatraat 
Groningen 
NETHERLANDS 

1  Dr.  Robert  Linn 

College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Frederick  M.  Lord 

Educational  Testing  Service 
Princeton,  NJ  085*0 

1  Dr.  James  Lumsden 

Department  of  Psychology 
University  of  Western  Australia 
Nedlands  W.A.  6009 
AUSTRALIA 

1  Dr.  Gary  Marco 

Educational  Testing  Service 
Princeton.  NJ  08*50 

1  Dr.  Scott  Maxwell 

Department  of  Psychology 
University  of  Houston 
Houston,  TX  7700* 

1  Dr.  Samuel  T.  Mayo 

Loyola  University  of  Chicago 
820  North  Michigan  Avenue 
Chicago,  IL  60611 

1  Dr.  Mark  Miller 

Computer  Science  Laboratory 
Texas  Instrunents ,  Inc. 

Hall  Station  371.  P.0.  Box  225936 
Dallas.  TX  75265 

1  Professor  Jason  Mlllman 
Department  of  Education 
Stone  Hall 
Cornell  University 
Ithaca,  NY  1*853 

1  Dr.  Allen  Munro 

Behavioral  Technology  Laboratories 
18*5  Elena  Ave.,  Fourth  Floor 
Redondo  Beach,  CA  90277 

1  Or  .  Donald  A  Norman 

Dept,  of  Psychology  C-009 
Unlv.  of  California.  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Melvin  R.  Novick 

356  Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City,  IA  522*2 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
*00  Army  Navy  Drive 
Arlington.  VA  22202 


Non  Govt 


1  Or.  Seynour  A.  Papert 

Massachusetts  Institute  of  Technology 
Artificial  Intelligence  Lab 
545  Technology  Square 
Cambridge,  HA  02139 

1  Or.  James  A.  Paulson 

Portland  State  University 
P.0.  Box  751 
Portland.  OH  97207 

1  MR.  LUIGI  PETRULLO 

2431  N.  EDGEWOOD  STREET 
ARLINGTON.  VA  22207 

1  DR.  PETER  POLSOII 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
BOULDER.  CO  80309 

I  DR.  DrANE  M.  RANSEY-KLEE 

R-K  RESEARCH  A  SYSTEM  DESIGN 
3947  RIDGEMCNT  DRIVE 
MALIBU,  CA  90265 

1  M  INRAT  M.  L.  RAUCH 
P  II  4 

BUNDESMINISTERIUM  DER  VERTEIDIGUNG 

POSTFACH  1328 

D-53  BONN  1,  CERMANY 

1  Dr.  Mark  D.  Reckas- 

Educational  Psychology  Dept. 
University  of  Mlssour UColunbla 
4  Hill  Hall 
Columbia,  MO  65211 

1  Dr.  Fred  Re  If 
SESAME 

c/o  Physics  Department 
University  of  California 
Berkely ,  CA  94720 

1  Dr.  Andrew  M.  Hose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  W 
Washington,  DC  20007 

1  Dr.  Leonard  L.  Rosenbaum,  Chairman 
Department  of  Psychology 
Montgomery  College 
Rockville.  HD  20S50 

1  Dr,  Ernst  Z.  Rothkopf 
Bell  Laboratories 
600  Mountain  Avenue 
Murray  Hill,  NJ  07974 

1  Or.  Lawrence  Rudner 
403  Elm  Avenue 
Takoma  Park.  MD  20012 

1  Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia.  SC  29208 

1  PROT.  FUNW>  SAMMIHA 
DEPT,  m  P3KMBWC* 

UNIVERSITY  OF  TENNESSEE 
KNOXVILLE.  TN  379)6 

1  Dr.  Alan  Sehoenfeld 

Department  of  Mathematics 
Hamilton  College 
Clinton,  NY  U323 


1  DR.  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUM  RRO 

300  N.  WASHINGTON  ST. 

ALEXANDRIA.  VA  22314 

1  Committee  on  Cognitive  Research 
t  Dr.  Lonnie  R.  Sherrod 
Social  Science  Research  Council 
605  Third  Avenue 
New  York,  NY  10016 

1  Or.  JCa 2uo  Shlgemasu 
University  of  Tohoku 
Department  of  Educational  Psychology 
Kawauchl ,  Sendai  980 

JAPAN 

1  Dr.  Edwin  Shirkey 

Department  of  Psychology 
University  of  Central  Florida 
Orlando,  FL  32816 

1  Dr.  Robert  Snlth 

Department  of  Computer  Science 

Rutgers  University 
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